Return of the Nuggets – AI Engineering

Charles F. Vardeman II

Center for Research Computing, University of Notre Dame

2023-11-17

This is all Peter’s falt!

Peter: How do we prompt better?

Prompting in the context of AI Engineering

AI Engineering as a Practice?

The Rise of the AI Engineer- by swyx - Laintent Space

AI Engineering Summit

AI Engineering Summit

AI Engineering Summit – Youtube

AI Engineering Summit

LLMOps Engineering

A Survey of Techniques for Maximizing LLM Performance

LLMOps Engineering

A Survey of Techniques for Maximizing LLM Performance

LLM Engineering – Prompt Engineering, RAG Engineering, Fine Tuning Engineering.

LLMOps – Cognitive Agents

Wang, Lei, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, et al. 2023. “A Survey on Large Language Model Based Autonomous Agents.” arXiv. http://arxiv.org/abs/2308.11432.

A Caution: Outward facing Data Fabric vs Inward Facing…

  • We don’t want to be engineering data silos!
  • With Agents, the “World Wide Web” is a Data Fabric!
  • We want to expose some information as Distributed Knowledge Graphs!

Data Fabrics are going to be used as Data Engines!

Twitter: Andrej Karpathy

How to Prompt?

How to Prompt Engineer…

Prompting Guide

Prompt Engineering Guide

OpenAI Cookbook

OpenAI Cookbook

Important: Prompt Structure Performance Changes with Model!

Prompt Testing?

Retrevial Augmented Generation

Unit Testing of LLMs

A Survey of Techniques for Maximizing LLM Performance

Prompt Engineering is about adding context!

KGs for Context

Sequeda, Juan, Dean Allemang, and Bryon Jacob. 2023. “A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model’s Accuracy for Question Answering on Enterprise SQL Databases.” arXiv. http://arxiv.org/abs/2311.07509.

KGs for Context

Sequeda, Juan, Dean Allemang, and Bryon Jacob. 2023. “A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model’s Accuracy for Question Answering on Enterprise SQL Databases.” arXiv. http://arxiv.org/abs/2311.07509.

Information Extraction for RAG (Tool Use)

Xu, Silei, Shicheng Liu, Theo Culhane, Elizaveta Pertseva, Meng-Hsi Wu, Sina J. Semnani, and Monica S. Lam. 2023. “Fine-Tuned LLMs Know More, Hallucinate Less with Few-Shot Sequence-to-Sequence Semantic Parsing over Wikidata.” arXiv. http://arxiv.org/abs/2305.14202.

Prompting Patterns for RAG – Planning and Action

Prasad, Archiki, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot. 2023. “ADaPT: As-Needed Decomposition and Planning with Language Models.” arXiv. https://doi.org/10.48550/arXiv.2311.05772.

Training and Fine Tuning

Textbooks are all you need II: phi-1.5

Y. Li, S. Bubeck, R. Eldan, A. Del Giorno, S. Gunasekar, and Y. T. Lee, “Textbooks Are All You Need II: phi-1.5 technical report.” arXiv, Sep. 11, 2023. Accessed: Sep. 12, 2023. [Online]. Available: http://arxiv.org/abs/2309.05463

Maybe we need more than Textbooks?

A “Curriculum” for Logic?

Feng, Jiazhan, Ruochen Xu, Junheng Hao, Hiteshi Sharma, Yelong Shen, Dongyan Zhao, and Weizhu Chen. 2023. “Language Models Can Be Logical Solvers.” arXiv. http://arxiv.org/abs/2311.06158.

A “Curriculum” for Logic?

Feng, Jiazhan, Ruochen Xu, Junheng Hao, Hiteshi Sharma, Yelong Shen, Dongyan Zhao, and Weizhu Chen. 2023. “Language Models Can Be Logical Solvers.” arXiv. http://arxiv.org/abs/2311.06158.